Choosing image labels

ABSTRACT

Methods, systems and apparatus for choosing image labels. In one aspect, a method includes receiving data specifying a first image, receiving text labels for the first image, receiving search results in response to a web search performed using at least some of the text labels as queries, ranking the text labels, at least in part, based on a number of resources referenced by the received search results, wherein at least some of the resources each include an image matching the first image, and selecting an image label for the image from the ranked text labels, the image label being selected based on the ranking.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of, and claims priorityto, U.S. patent application Ser. No. 13/486,167, titled “Choosing ImageLabels,” filed on Jun. 1, 2012. The disclosure of the foregoingapplication is incorporated herein by reference in its entirety for allpurposes.

BACKGROUND

This specification relates to data processing and choosing image labels.

A search processing operation that provides search results responsive toa query image can provide information of interest to a user about thequery image. For example, the search processing operation can findinformation available on the Internet describing the image, such as anorigin, creator, or artist of an image; other versions of the image suchas cropped, edited, or different resolution versions of the image;companies, organizations, or other entities related to the image, e.g.,if the image is a logo, news stories related to the image; or identitiesof subjects of an image such as buildings or objects portrayed in theimage. One way to find information available on the Internet describingthe image is to perform a text search using as a query a text labeldescribing the image. An image annotator can be used to identify textlabels describing the image.

SUMMARY

In general, in one aspect, a method includes receiving data specifying afirst image, receiving text labels for the first image, receiving searchresults in response to a web search performed using at least some of thetext labels as queries, ranking the text labels, at least in part, basedon a number of resources referenced by the received search results,wherein at least some of the resources each include an image matchingthe first image, and selecting an image label for the image from theranked text labels, the image label being selected based on the ranking.

Implementations of this aspect may include one or more of the followingfeatures. Receiving text labels for the first image includes requestingtext labels associated with images identified as near duplicate imagesof the first image. The aspect includes ranking the received text labelsby assigning a score to each text label and incrementing the score foreach instance of another text label that is identified as being a nearduplicate text label of the text label. The resources matching the firstimage include resources containing an image identified as a nearduplicate image of the first image. The resources matching the firstimage include resources containing an image identified as residing in acluster of a database containing the first image. The aspect includesoutputting a highest ranked text label as responsive to the first image.The data specifying the first image is received from a user device, andthe received text labels are identified based on characteristics of auser of the user device. Ranking the text labels includes assigning atext label ranking score to each of the text labels, and arranging thetext labels according to the assigned score. Ranking the text labelsincludes calculating a text label ranking score for each text labelbased on a median search result score calculated according to searchresult scores associated with the search results, a number of searchresults returned by the web search, and an original rank position of thereceived text labels. Ranking the text labels includes calculating atext label ranking score for each text label based on the formula:median_score*log(1+max(1,docs_matched))*smoothing_factor/(smoothing_factor+original_rank_position),wherein median_score specifies a median search result score calculatedaccording to search result scores associated with the search results,docs_matched specifies a number of search results returned by the websearch, smoothing_factor is a constant value, and original_rank_positionspecifies an original rank position of the received text labels.

Particular implementations of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. A text label can be chosen as a descriptor for animage. The text label can be used to perform a web search for contentrelevant to the image.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features,aspects, and advantages will be apparent from the description anddrawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an example environment in which a searchsystem provides search services.

FIG. 2 is a block diagram showing an example data flow for selecting alabel that describes a query image.

FIG. 3 is an example spatial representation of a portion of an imagedatabase.

FIG. 4 is a flowchart of an example process for choosing a label for animage.

FIG. 5 is a block diagram of an example computer system.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

One way that a search processing operation can find information relatedto a query image is to identify a text label describing the query imageand perform a textual web search using the text label as a query. Forexample, the search processing operation could maintain a database thatassociates images with text labels and use the database to identify atext label that matches the query image. This database could alsoassociate images with other images by clustering the images togetheraccording to visual features common to the images.

The text label that best matches the query image can be identified bycompiling a list of text labels that are associated with images thathave been determined to be visually similar to the query image. Then,the list can be re-ordered according to which text labels occur mostfrequently in this list, so that the most likely candidates for the besttext label are at the top of the list. Next, the search processingoperation can perform a web search using the candidates for the besttext label as queries and pare down or filter the search results basedon whether the search results include web pages that contain images thathave been identified as near-duplicate images to the query image, orimages clustered to a same group as the near-duplicate images within thedatabase of images. The list of candidates for the best text label canbe re-ordered again based on how many pared-down search results werereturned for that label. The text label at the top of the re-orderedlist can be selected as a top candidate for the text label that bestmatches the query image.

FIG. 1 is a block diagram of an example environment 100 in which asearch system 110 provides search services. The example environment 100includes a network 102, e.g., a local area network (LAN), wide areanetwork (WAN), the Internet, or a combination of them, connects websites 104, user devices 106, and the search system 110. The environment100 may include many web sites 104 and user devices 106, which are alsosometimes referred to as client devices.

A web site 104 is a facility containing one or more resources 105associated with a domain name and hosted by one or more servers. Anexample web site is a collection of web pages formatted in hypertextmarkup language (HTML) that can contain text, images, multimediacontent, and programming elements, e.g., scripts. Each web site 104 ismaintained by a publisher, e.g., an entity that manages and/or owns theweb site.

A resource 105 is data that can be provided by the web site 104 over thenetwork 102 and that is associated with a resource address. Resources105 include HTML pages, word processing documents, and portable documentformat (PDF) documents, images, video, and feed sources, to name just afew. The resources can include content, e.g., words, phrases, images andsounds and may include embedded information, e.g., meta information andhyperlinks, and/or embedded instructions, e.g., scripts. In someexamples, one resource 105 can contain or reference another resource105. For example, an HTML page can reference an image.

A user device 106 is an electronic device that is under control of auser and is capable of requesting and receiving resources over thenetwork 102. Example user devices 106 include personal computers, mobilecommunication devices, and other devices that can send and receive dataover the network 102. A user device 106 typically includes a userapplication, e.g., a web browser, to facilitate the sending andreceiving of data over the network 102.

To facilitate searching of resources, the search system 110 identifiesthe resources by crawling and indexing the resources 105 provided by theweb sites 104. Data about the resources 105 can be indexed based on theresource 105 to which the data corresponds.

The user devices 106 submit search queries 109 to the search system 110.In response, the search system 110 identifies resources 105 that areresponsive to, e.g., have at least a threshold relevance score for, thesearch query 109. The search system 110 generates search results 111that identify the resources 105 and returns the search results 111 tothe user devices 106. A search result 111 is data generated by thesearch system 110 that identifies a resource 105 that is responsive to aparticular search query, and includes a link to the resource 105. Anexample search result 111 can include a web page title, a snippet oftext or a portion of an image extracted from the web page, and the URLof the web page. For example, the search queries 109 can be submittedduring user sessions in which a user of a user device 106 enters searchqueries into a user interface. During a user session, the user can bepresented with search results 111.

The user devices 106 can receive the search results 111, e.g., in theform of one or more web pages, and render the web pages for presentationto users. In response to the user selecting a link in a search result111 at a user device 106, the user device 106 requests the resource 105identified by the link. The web site 104 hosting the resource 105receives the request for the resource 105 from the user device 106 andprovides the resource 105 to the requesting user device 106. In someexamples, the search system 110 can also generate search queries 109 forwhich search results 111 are generated. For example, the search system110 can receive a search query 109 that includes an image. The searchsystem 110 can then determine a text label that has been identified asrelevant to the image, and use the text label as a search query 109 toreceive search results 111 relevant to the image.

Search results 111 can be ranked based on scores related to theresources 105 identified by the search results 111, such as informationretrieval (“IR”) scores, and optionally a quality score of each resourcerelative to other resources 105. In some implementations, the IR scoresare computed from dot products of a feature vector corresponding to asearch query 109 and feature vectors of resources 105, and the rankingof the search results 111 is based on relevance scores that are acombination, e.g., sums, products, or other mathematical combinations,of the IR scores and quality scores. In some examples, the searchresults 111 can be ordered at least partially according to theserelevance scores and provided to the user device according to the order.

In some implementations, a search query 109 can include data for asingle query type or for two or more query types, e.g., types of data inthe query. For example, the search query 109 may have a text portion,and the search query 109 may also have an image portion. A search query109 that includes data for two or more query types can be referred to asa “hybrid query.” In some examples, a search query 109 includes data foronly one type of query. For example, the search query 109 may onlyinclude image query data, e.g., a query image, or the search query mayonly include textual query data, e.g., a text query.

In some examples, the search query 109 includes a query image 113. Insome implementations, the search system 110 returns search results 111responsive to the query image 113. For example, the search results 111could be responsive to a textual query describing the query image 113.

In some implementations, the search system 110 includes a text searchapparatus 130 that is used to perform a search based on a textual input.For example, the text search apparatus 130 can perform a search based ona label for an image, which is referred to as an image label.

An image label (“label”) is data that is indicative of subject matter towhich an image is relevant. Labels can be explicitly specified by apublisher of a web site 104 on which the image appears. Labels can alsobe generated, for example, based on text that appears near the image onthe web page. For example, a label can be generated for an image basedon text that is located adjacent to, e.g., within a threshold number ofpixels of, the image or in a portion of a document that is identified ascontaining information relevant to the image, e.g., a frame in which theimage is presented. A label can also be generated based on text that isincluded in the image, e.g., visually depicted in the image, orotherwise associated with the image, e.g., text included in a file name,text included in anchor text of a link to the image, or resourcelocation of the image.

An image label apparatus 126, sometimes called an image annotator, is adata processing apparatus that can generate labels for images. The imagelabel apparatus 126 can receive an image such as a query image 113 asinput. The image label apparatus 126 can return one or more labels thatdescribe a topic of or are otherwise semantically related to the image.In some implementations, the image label apparatus 126 can identify alabel that is semantically related to an image because the image labelapparatus 126 may already store data describing the image and/or dataindicating which labels are semantically related to the image. The imagelabel apparatus 126 can identify a label that is semantically related toan image because the image label apparatus 126 may store dataassociating visual features of images with labels.

In some examples, the image label apparatus 126 can interact with animage relevance model 112 to identify a label for an image based onvisual features of the image. The image label apparatus 126 can providethe image to an image relevance model 112 that has been created for aparticular label to determine the degree to which the image is relevantto the particular label. When the image relevance model 112 receives animage as input, the image relevance model 112 can output a relevancescore indicating the degree to which the input image is related to theparticular label corresponding to the image relevance model 112.

In some examples, the environment 100 also includes an image similarityapparatus 124. The image similarity apparatus 124 can receive an imageas input and identify other images that are visually similar to theimage. In some implementations, images can be identified as visuallysimilar if the images have similar sets of visual features, such assimilar colors, brightness, shapes, edge locations, and/or other similarattributes. For example, images of a sunrise will likely share similarcolors, e.g., of a sky at sunrise, and shapes, e.g., of a sun appearingon the horizon. The attributes identified by the image relevance model112 can be used to further identify other images sharing the same commonfeatures. The image similarity apparatus 124 may include or be incommunication with an image database 128 that contains information aboutimages and stores, information identifying images that are included inimage clusters. For example, the image database 128 may cluster togetherimages having similar sets of visual features such as near duplicateimages.

In some examples, the images in the image database 128 are clusteredbased on their content feature values, i.e., values indicative of visualfeatures of the image and/or other image features. The content featurevalues can be extracted for each image represented in the image database128. The content feature values can be transformed into a sparserepresentation using a pre-computed dictionary of visual terms thatrepresent a set of regions having similar content feature values. Theextraction and transformation yields a “bag of features” for the image.Generally, the content feature values of an image quantitativelyrepresent visual features of the image. Because the content featurevalues are represented as numerical values, a mathematical calculationcan be performed to determine a degree of similarity between contentfeature values of two images. Two images having content feature valuesmeeting a threshold degree of similarity can be clustered together inthe image database 128. Note that other techniques can be used tocluster images that are considered visually similar.

The image similarity apparatus 124 can be configured to identify imagesthat are near duplicate images relative to an image received by theimage similarity apparatus 124 as input. In some implementations, nearduplicate images are images that are identical except for differences inthe way the images have been processed. In some examples, thedifferences between two near duplicate images are differences other thandifferences in visual features of the images. As one example, twoidentical images at different resolutions are near duplicate images. Asanother example, two images of the same object having differentbrightness, contrast, or other image attributes, e.g., image attributesadjusted in image processing software, can be near duplicate images. Asanother example, an uncropped version of an image and a cropped versionof the same image, e.g., cropped to remove a border in the image, can benear duplicate images. Images can be identified as near duplicate imagesif they satisfy a similarity threshold, e.g., a similarity scoredetermined by the image similarity apparatus 124 that exceeds asimilarity threshold.

The image similarity apparatus 124 can be used to identify images of aresource 105, for example, images contained in or referenced by a webpage, which match an image received as a portion of a search query 109.An image contained in a web page can be said to match a query image 113if the images are near duplicate images according to the imagesimilarity apparatus 124.

FIG. 2 is a block diagram showing an example data flow for selecting alabel 202 that describes a query image 113. For example, the label thatdescribes the query image 113 can be the label that best describes thequery image 113. Here, a label that best describes an image is one that,when used as a textual search query, returns results most relevant, orhaving at least a threshold measure of relevance, to the query image113. Put another way, the best label can be a label that is consideredto have at least a threshold likelihood of, e.g., most likely to,accurately describing the query image 113. As shown in FIG. 2, thesearch system 110 interacts with related components to choose a textlabel. In some implementations, components other than the componentsshown here can undertake the same or similar operations to choose a textlabel.

The search system 110 provides the query image 113 to an image labelapparatus 126. For example, the query image 113 may have been receivedas a portion of a search query 109 submitted by a user device 106 asshown in FIG. 1. The image label apparatus 126 returns labels 204 thatare considered descriptive of the query image 113. For example, if thequery image 113 is supplied to an image relevance model 112 (FIG. 1)corresponding to the label 204 and the image relevance model 112 returnsan image relevance score greater than a threshold, then the label 204can be said to describe the query image 113.

In some implementations, the labels 204 are selected by the image labelapparatus 126 based on visual features of the query image 113. In someexamples, the labels 204 may each correspond to image relevance modelsthat weight visual features based on the corresponding label, so thatvisual features found in images identified as matching the label areweighed higher in the image relevance model. The image label apparatus126 may have identified those same, or similar, visual features in thequery image 113. In some examples, the label 204 may be associated withthe particular query image 113 in data stored by the image labelapparatus 126 associating images with labels.

In some examples, the labels 204 are ranked by the image label apparatus126. For example, the image label apparatus 126 may provide the labels204 in the form of an ordered list, with the label most likely toaccurately describe the query image 113 at the highest position in thelist and the label least likely to accurately describe the query image113 at the lowest position in the list. The labels 204 are provided tothe search system 110.

In some examples, the labels 204 are chosen based on characteristics ofa user of the user device 106 (FIG. 1) who submitted the query image113. For example, if the user is identified as speaking a particularlanguage, labels 204 can be chosen in the user's language. In someexamples, the user indicates that he prefers a particular language, forexample, in a user profile of the user stored by the search system 110.In some examples, the user may be identified as speaking a particularlanguage based on a location of the user, for example, an indication bythe user that he or she resides in a particular country. Othercharacteristics of a user, for example, characteristics specified by theuser, can be used to select the labels 204.

In some implementations, the search system 110 also provides the queryimage 113 to the image similarity apparatus 124. The image similarityapparatus 124 returns near duplicate images 208 of the query image 113.In some examples, the image similarity apparatus 124 returns informationdescribing the near duplicate images 208, such as image identifiers orreferences to the near duplicate images 208. The near duplicate images208 may be images that have some or all of the same visual features asthe query image 113. In some examples, the near duplicate images 208 canalso be provided to the image label apparatus 126. In response, theimage label apparatus 126 can identify labels 206 describing the nearduplicate images 208 and return those labels 206 to the search system110.

If the search system 110 receives both sets of labels 204, 206, thesearch system 110 can merge the two sets into a single merged set oflabels 210. In some examples, some of the labels 204 describing thequery images 113 also appear among the labels 206 describing the nearduplicate images 208. When the search system 110 merges the two sets oflabels 204, 206, the search system 110 can determine the number ofoccurrences of the same label among the sets of labels 204, 206. Upongenerating a merged set of labels 210, the search system can assignlabel scores 212 to each of the labels 210. The label score 212 of eachlabel 210 represents the number of times a particular label appeared inthe sets of labels 204, 206. For example, if a particular label wasreturned for three near duplicate images 208 then that particular labelmay be assigned a score of three.

In some examples, the merged set of labels 210 will include labels thatare near duplicates of each other. Two labels are near duplicates ofeach other if the labels meet a similarity condition. One example of asimilarity condition is whether two labels differ by less than athreshold quantity of characters. For example, if the threshold quantityof characters is one, then the label “cat” is a near duplicate label tothe label “cats.”

Another example of a similarity condition is whether two labels areidentical except for one or more identified stop words. A stop word is aword that appears often in labels but does not change the semanticmeaning of the label. For example, stop words may include words such asprepositions or articles, or words that appear often in labels such as“pictures” or “images.” Thus, the labels “cats,” “pictures of cats,” and“cats images” may all be near duplicate labels of each other.

The search system 110 can remove near duplicate labels in the merged setof labels 210, leaving one label representing the other near duplicatelabels, and increment the label score 212 associated with the remaininglabel by the number of near duplicate labels removed. Thus, if thelabels “pictures of cats,” and “cats images” are removed, then the labelscore 212 associated with the label “cats” can be incremented by two.Although in the example described here the label score 212 is firstcalculated based on occurrences of the same label being returned fornear duplicate images, and then incremented based on near duplicatelabels, in some examples, the label score 212 is first calculated basedon near duplicate labels, and then incremented based occurrences of thesame label being returned for near duplicate images. Once the labelscore 212 is calculated for each label, the merged set of labels 210 canbe re-ranked. For example, if the set of labels 210 is an ordered list,the set of labels 210 can be re-ranked so that labels having a higherlabel score appear higher in the list and labels having a lower labelscore appear lower in the list.

The search system 110 then provides the merged set of labels 210 to thetext search apparatus 130 as text queries. The text search apparatus 130performs a search of online resources 105 (FIG. 1) using the set oflabels 210. For example, the online resources 105 can include web pagesresponsive to a particular label. The text search apparatus 130 thenprovides search results 111 back to the search system 110.

In some implementations, the search results 111 can be filtered based onthe resources 105 referenced by the search results 111. In someexamples, the search results 111 can be filtered to include only thosesearch results that reference resources 105 containing the query image113 and/or one of the near duplicate images 208. For example, if aparticular web page referenced by the search results 111 does notcontain the query image 113 or one of the near duplicate images 208 thenthe search result referencing that web page can be excluded from thesearch results 111.

In some examples, the search results 111 can be filtered to referenceresources 105 containing an image belonging to the same cluster as thequery image 113 in an image database. Image clusters are described inmore detail in FIG. 3. The search results 111 can be filtered by thesearch system 110, the text search apparatus 130, or another apparatus.In this way, the search results 111 can be filtered such that theresources 105 referenced by the search results 111 are more likely to berelevant to the query image 113 than if the search results 111 were notfiltered.

The search system 110 can use the search results 111 to re-order themerged set of labels 210. In some examples, the search results 111 areeach associated with a search result score 214. A search result score214 is a numerical value that indicates the degree to which a particularsearch result is relevant to the original search query. In someimplementations, the search system could calculate a median searchresult score 214 and order the merged set of labels 210 according to themedian search result score 214 for each label 210. In someimplementations, the search system could order the merged set of labels210 according to the total number of search results 111 returned foreach label 210.

In some examples, the search system 110 assigns a ranking score 216 toeach label in the merged set of labels 210. A ranking score 216 is anumerical value that can be calculated by the search system 110 based onmeasures indicative of the degree to which a label describes an image.In this way, the ranking score 216 for a label indicates how well thelabel describes the image. A label in the set of labels 210 having thehighest ranking score 216 can be said to be the label that bestdescribes the image corresponding to the label.

The ranking score 216 can be calculated based on multiple variables. Insome examples, the ranking score 216 is calculated based on the mediansearch result score calculated according to search result scores 214associated with the search results 111 for a particular label 210. Insome examples, the ranking score 216 is calculated based on the numberof search results 111 returned for a particular label 210. In someexamples, the ranking score 216 is calculated based on the original rankposition of the labels 210, for example, as the labels were rankedaccording to label scores 212.

In some implementations, the ranking score 216 for a label 210 can becalculated based on the following equation, in which median_scorespecifies the median search result score calculated according to searchresult scores 214 associated with the search results 111 returned forthis label 210, docs_matched specifies a number of search results 111returned by the web search carried out for this label 210, which may befiltered search results 111, smoothing_factor is a constant value chosenbased on experimental data, and original_rank_position specifies anoriginal rank position of the labels 210:

${ranking\_ score} = \frac{\begin{matrix}{{median\_ score} \times \log \left( {1 + {\max \left( {1,{docs\_ matched}} \right)}} \right) \times} \\{smoothing\_ factor}\end{matrix}}{{smoothing\_ factor} + {{original\_ rank}{\_ position}}}$

Once the ranking score 216 is calculated for each label in the mergedset of labels 210, the label 202 having the highest ranking score 216can be chosen as most responsive to the query image 113, or, put anotherway, the label 202 that best describes the query image 113.

FIG. 3 is an example spatial representation 300 of visual distancesbetween images in a portion of an image database. For example, the imagedatabase could be the image database 128 shown in FIG. 1. The portion ofthe image database shown includes five images 302, 304, 306, 308, 310.In this example spatial representation 300, images are closer to eachother if they share more visual features in common, and farther indistance from each other if they share fewer visual features in common.Thus, the relative distance between two images is a representation ofhow many visual features they share in common, relative to other images.

Images that share a threshold quantity of visual features in common canbe said to be in a same cluster. In the example spatial representation300 images that are separated by less than a chosen spatial distance arewithin a same cluster. For example, one image 302 is closer to someimages and farther away from other images. A distance 312 from the image302 can be chosen as the threshold between images clustered with theimage 302 and images not clustered with the image 302. Images within acircle 314 defined by the distance 312 are clustered in a same clusterwith the image 302. Put another way, the circle 314 defines a cluster ofimages based on the first image 302. One image 304 is within the circle314 because it is a distance 316 from the first image 302 less than thethreshold distance 312. Thus, that image 304 is clustered with the firstimage 302. Another image 308 is not within the circle 314 because it isa distance 318 from the first image 302 greater than the thresholddistance 312. Thus, that image 308 is not clustered with the first image302.

Because images 304, 306 that are clustered with the first image 302share a threshold quantity of visual features in common with the firstimage 302, the images 304, 306 may represent similar subjects as thefirst image 302. For example, the images 304, 306 may portray the samebuildings, objects, or other subjects. As described above with respectto FIG. 2, search results 111 can be filtered to only include resources105 that contain an image 302, for example, a query image 113, and otherimages 304, 306 clustered with that image. In this way, the searchresults 111 will be filtered to include resources 105 that are likely tobe relevant to the image 302.

FIG. 4 is a flowchart of an example process 400 for choosing a label foran image. The operations of the process 400 can be performed, forexample, by a search system 110 (FIG. 1).

At operation 402, data specifying an image is received. For example, theimage could be an image submitted as part of a search query 109 (FIG.1). The data could be, for example, a location of the image such as auniform resource locator pointing to a location at which the image isstored. In some examples, the data could be the image itself arranged ina file format for images.

At operation 404, text labels associated with the specified image arereceived. For example, the specified image may be provided to an imagelabel apparatus 126 (FIG. 1) that returns labels identified as relevantto the image. In some examples, the labels are identified as relevant toan image that is a near duplicate image of the specified image. In someimplementations, the text labels may be ranked in an order based on therelevance of the text label to the specified image.

At operation 406, data is received in response to a web search. The websearch can be performed using the received text labels as individualsearch queries. For example, a text search apparatus 130 (FIG. 1) cancarry out the search. The data may be in the form of search results 111referencing resources 105 determined to be relevant to a correspondingsearch query. In some examples, the search results 111 can be filteredso that the referenced resources 105 are relevant to the specifiedimage. For example, resources 105 matching the specified image may beidentified, and resources 105 not matching the specified image can beremoved. A resource 105 may match the specified image if the resourcecontains the specified image, or the resource contains a near duplicateimage of the specified image, or the resource 105 contains an imageclustered with the specified image in an image database.

At operation 408, the text labels are ranked. The text labels can beranked based on a number of resources referenced by the received datathat include an image matching the specified image. In some examples, animage matches the specified image if the two images contain identicaldata or at least a threshold visual similarity. In some examples, animage matches the specified image if the images are near duplicates ofeach other. In some examples, an image matches the specified image ifthe two images are in the same cluster. In some implementations, theresources can be resources referenced by the search results. In someexamples, the text labels can be scored and ranked based on their score.For example, the score can be calculated based on variables such as amedian search result score calculated according to search result scoresassociated with the search results for a particular label, or based onthe number of search results returned for a particular label, or basedon an original rank position of the text labels.

At operation 410, an image label for the image can be selected from theranked text labels. The image label is selected based on the ranking.For example, the image label can be the highest ranked text label. Thehighest ranked text label can be said to be the text label determined tobe most relevant to the specified image. The selected image label can beoutput as the text label responsive to the specified image. The selectedimage label can be used as a text query to perform a text-based search,and the results of this search can be returned as responsive to a searchperformed using the specified image as a query image.

FIG. 5 is a block diagram of an example computer system 500. Forexample, the system 500 could be a system or a portion of a systemexecuting the search system 110 or other systems shown in FIG. 1. Thesystem 500 includes a processor 510, a memory 520, a storage device 530,and an input/output device 540. Each of the components 510, 520, 530,and 540 can be interconnected, for example, using a system bus 550. Theprocessor 510 is capable of processing instructions for execution withinthe system 500. In one implementation, the processor 510 is asingle-threaded processor. In another implementation, the processor 510is a multi-threaded processor. The processor 510 is capable ofprocessing instructions stored in the memory 520 or on the storagedevice 530.

The memory 520 stores information within the system 500. In oneimplementation, the memory 520 is a computer-readable medium. In oneimplementation, the memory 520 is a volatile memory unit. In anotherimplementation, the memory 520 is a non-volatile memory unit.

The storage device 530 is capable of providing mass storage for thesystem 500. In one implementation, the storage device 530 is acomputer-readable medium. In various different implementations, thestorage device 530 can include, for example, a hard disk device, anoptical disk device, or some other large capacity storage device.

The input/output device 540 provides input/output operations for thesystem 500. In one implementation, the input/output device 540 caninclude one or more of a network interface devices, e.g., an Ethernetcard, a serial communication device, e.g., an RS-232 port, and/or awireless interface device, e.g., and 802.11 card. In anotherimplementation, the input/output device can include driver devicesconfigured to receive input data and send output data to otherinput/output devices, e.g., keyboard, printer and display devices 560.Other implementations, however, can also be used, such as mobilecomputing devices, mobile communication devices, set-top box televisionclient devices, etc.

Implementations of the subject matter and the operations described inthis specification can be implemented in digital electronic circuitry,or in computer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Implementations of the subjectmatter described in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on a computer storage medium for execution by, orto control the operation of, data processing apparatus. Alternatively orin addition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media, e.g., multiple CDs, disks, orother storage devices.

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing. The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program, also known as a program, software, softwareapplication, script, or code, can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data, e.g., one or more scripts stored in a markup language document,in a single file dedicated to the program in question, or in multiplecoordinated files, e.g., files that store one or more modules,sub-programs, or portions of code. A computer program can be deployed tobe executed on one computer or on multiple computers that are located atone site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storagedevice, e.g., a universal serial bus (USB) flash drive, to name just afew. Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Implementations of the subject matter described in this specificationcan be implemented in a computing system that includes a back-endcomponent, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front-endcomponent, e.g., a client computer having a graphical user interface ora Web browser through which a user can interact with an implementationof the subject matter described in this specification, or anycombination of one or more such back-end, middleware, or front-endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. Examples of communication networks include a local areanetwork, “LAN”, and a wide area network, “WAN”, an inter-network, e.g.,the Internet, and peer-to-peer networks, e.g., ad hoc peer-to-peernetworks.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someimplementations, a server transmits data, e.g., an HTML page, to aclient device, e.g., for purposes of displaying data to and receivinguser input from a user interacting with the client device. Datagenerated at the client device, e.g., a result of the user interaction,can be received from the client device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope or of what maybe claimed, but rather as descriptions of features specific toparticular implementations. Certain features that are described in thisspecification in the context of separate implementations can also beimplemented in combination in a single implementation. Conversely,various features that are described in the context of a singleimplementation can also be implemented in multiple implementationsseparately or in any suitable subcombination. Moreover, althoughfeatures may be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and theclaimed combination may be directed to a subcombination or variation ofa subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular implementations have been described. Otherimplementations are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

1. (canceled)
 2. A computer-implemented method comprising: receivingdata specifying a first image; receiving text labels for the firstimage; for each of one or more of the text labels: receiving datadescribing a set of web resources that each have at least a thresholdrelevance score that measures the relevance of the web resource to thetext label; assigning a text label ranking score to the text label basedon a number of the resources in the set of web resources that include animage matching the first image; ranking the text labels, at least inpart, based on the text label ranking scores assigned to text labels;and selecting an image label for the first image from the ranked textlabels, the image label being selected based on the ranking.
 3. Thecomputer-implemented method of claim 2, wherein receiving text labelsfor the first image includes requesting text labels associated withimages identified as near duplicate images of the first image.
 4. Thecomputer-implemented method of claim 2, wherein assigning a text labelranking score comprises incrementing the score for each instance ofanother text label that is identified as being a near duplicate textlabel of the text label.
 5. The computer-implemented method of claim 2,further comprising outputting a highest ranked text label as responsiveto the first image.
 6. The computer-implemented method of claim 2,wherein: the data specifying the first image is received from a userdevice, and the received text labels are identified based oncharacteristics of a user of the user device.
 7. Thecomputer-implemented method of claim 2, wherein ranking the text labelsincludes arranging the text labels according to the corresponding textlabel ranking scores.
 8. A system, comprising: a data processingapparatus; and a memory coupled to the data processing apparatus havinginstructions stored thereon which, when executed by the data processingapparatus cause the data processing apparatus to perform operationscomprising: receiving data specifying a first image; receiving textlabels for the first image; for each of one or more of the text labels:receiving data describing a set of web resources that each have at leasta threshold relevance score that measures the relevance of the webresource to the text label; assigning a text label ranking score to thetext label based on a number of the resources in the set of webresources that include an image matching the first image; ranking thetext labels, at least in part, based on the text label ranking scoresassigned to text labels; and selecting an image label for the firstimage from the ranked text labels, the image label being selected basedon the ranking.
 9. The system of claim 8, wherein receiving text labelsfor the first image includes requesting text labels associated withimages identified as near duplicate images of the first image.
 10. Thesystem of claim 8, wherein assigning a text label ranking scorecomprises incrementing the score for each instance of another text labelthat is identified as being a near duplicate text label of the textlabel.
 11. The system of claim 8, the operations further comprisingoutputting a highest ranked text label as responsive to the first image.12. The system of claim 8, wherein: the data specifying the first imageis received from a user device, and the received text labels areidentified based on characteristics of a user of the user device. 13.The system of claim 8, wherein ranking the text labels includesarranging the text labels according to the corresponding text labelranking scores.
 14. Non-transitory computer readable media storingsoftware comprising instructions executable by a processing device andupon such execution cause the processing device to perform operationscomprising: receiving data specifying a first image; receiving textlabels for the first image; for each of one or more of the text labels:receiving data describing a set of web resources that each have at leasta threshold relevance score that measures the relevance of the webresource to the text label; assigning a text label ranking score to thetext label based on a number of the resources in the set of webresources that include an image matching the first image; ranking thetext labels, at least in part, based on the text label ranking scoresassigned to text labels; and selecting an image label for the firstimage from the ranked text labels, the image label being selected basedon the ranking.